ronomon-queue

ronomon-queue

Process thousands of asynchronous or synchronous jobs, concurrently or sequentially, safely and efficiently, without creating thousands of closures.

Installation

ronomon-queue has no dependencies.

npm install ronomon-queue

Usage

var queue = new Queue([concurrency])

• concurrency An integer >= 1, the maximum number of jobs to run concurrently. Default: 1

Example:

var Queue = require('ronomon-queue');

var queue = new Queue(4);
queue.onData = function(integer, end) {
  console.log('Processing ' + integer + '...');
  // Notify the queue that we are done processing job:
  end();
};
queue.onEnd = function(error) {
  if (error) throw error;
  console.log('Done');
};
queue.push([1, 2, 3]);
queue.push(4);
queue.push([5, 6]);
queue.push(7);
queue.push([8, 9, 10]);
// Notify the queue that we are done pushing jobs:
queue.end();

queue.onData = function(job, end) { end() }

• job A job which was pushed individually or pushed as part of an Array of jobs.
• end A callback for onData() to call when onData() has finished processing job.

A function which must be provided and which will be called once for each job. If an error is encountered while processing the job, this can be passed to the end callback to stop the queue and return an error to the onEnd() function.

queue.onEnd = function([error]) {}

• error An error encountered (if any) while pushing jobs onto the queue, or while processing jobs in the queue.

A function which must be provided and which will be called once queue.end() has been called and once all running and pending jobs in the queue complete. If an error is encountered, then onEnd(error) will be called as soon as all running jobs complete. If the queue is stopped using stop(), then onEnd() will be called as soon as all running jobs complete.

queue.push(job)

• job Any object, will be passed to the onData() function.

queue.push(jobs)

• jobs An Array of jobs, each element of the array will be passed to the onData() function.

Pushing an Array of jobs is more efficient than calling push() for each job, as ronomon-queue will iterate the pushed array directly, without modifying it, and without copying the Array.

queue.end()

Notify the queue that no further jobs will be pushed. The queue will wait for all running and all pending jobs to complete, and will then call the onEnd() function. queue.end() is idempotent, successive calls will be ignored.

queue.end(error)

• error An error encountered while pushing jobs onto the queue.

Notify the queue that no further jobs will be pushed because an error was encountered while pushing jobs onto the queue. The queue will ignore any pending jobs, will wait for all running jobs to complete, and will then call the onEnd() function. The onEnd() function will be called with the same error, if no jobs return a different error before queue.end(error) is called. queue.end(error) is idempotent, successive calls will be ignored.

queue.stop([error])

• error An error encountered while pushing jobs onto the queue, or while processing jobs in the queue.

An optional method to notify the queue that any pending jobs should be ignored. The queue will ignore any pending jobs, will wait for all running jobs to complete, and will then call the onEnd() function. If no error argument is provided, then the onEnd() function will be called without an error, provided no jobs return an error before stop() is called (if any running jobs return an error after stop() is called, their errors will be ignored). stop() is idempotent, successive calls will be ignored.

There are several differences between stop() and queue.end(). stop() will ignore any pending jobs, whereas queue.end() will wait for all pending jobs to complete before calling the onEnd() function. A successive call to queue.end(error) will be ignored if queue.end() has already been called, whereas stop(error) will stop the queue even if queue.end() has already been called.

var pool = new Queue.Pool([concurrency])

• concurrency An integer >= 1, the maximum number of jobs to run concurrently across all queues. Default: 1

It is possible to limit the total concurrency across multiple queues as a whole, by assigning a shared instance of Queue.Pool to each queue.

Example:

// Each queue will execute at most 32 jobs concurrently.
// At most 64 jobs in total will be executed concurrently.
var pool = new Queue.Pool(64);
var queue1 = new Queue(32);
var queue2 = new Queue(32);
var queue3 = new Queue(32);
var queue4 = new Queue(32);
queue1.pool = pool;
queue2.pool = pool;
queue3.pool = pool;
queue4.pool = pool;

Tests

ronomon-queue ships with a long-running fuzz test:

node test.js

Benchmark

To benchmark the cost of the queue implementation:

node benchmark.js